Re-Visiting Backpropagation Network Optimization: Towards Maximally Pruned Networks

Backpropagation (BP) Neural Network (NN) error functions enable the mapping of data vectors to user-defined classifications by driving weight matrix modifications so as to reduce classification error over the training data set. Conventional BP error functions are usually only implicitly dependant on the weight matrix, however an explicit penalty term can be added so as to force numerically insignificant weights closer to zero. In our investigation, BP training is undertaken as a prelude to a pruning stage that selectively removes functionally unimportant weight matrix elements, thereby resulting in sparser network connectivity more suited for subsequent rule extraction. This paper investigates the usage of several error and activation functions in the effort to produce maximally clean network connections.